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Questionnaire Research in Higher Education 

Yukiko Inoue 
University of Guam 

Abstract 

The objective of this paper was to discuss a survey as a research method based on the three 
questionnaire surveys developed and administered in educational settings: Undergraduate Women 
Questionnaire Survey was conducted in 1998 to explore undergraduate women’s status aspiration and 
gender awareness; Computer-Assisted Instruction Questionnaire Survey was conducted in 1995 to 
identify and to prioritize the factors influencing the university faculty members' use of computers in 
their teaching; and Diversity and Multiculturalism Questionnaire Survey was conducted in 2000 to 
determine the attitudes of the university faculty toward multicultural education. These questionnaire 
surveys were developed for the different research purposes but revealed the common issues and 
concerns, in particular, in terms of designing surveys, measuring survey reliability and validity, and 
reporting on surveys. 



There are many ways to obtain research data from people. The most common methods are 
interviews (including focus groups and personal interviews) and surveys of various kinds 
(including telephone surveys, mail surveys, and e-mail surveys). In general, a survey can be 
defined as follows: “A survey is a system for collecting information to describe, compare, or 
explain knowledge, attitudes, and behavior” (Fink, 1995, p. 1). Survey questionnaires provide an 
efficient way to collect data. Questionnaires can: (i) reach large numbers of people at relatively 
low cost; (ii) ensure anonymity; and (iii) be written for specific purposes. The purpose of this 
paper is to discuss questionnaire research in higher education on the basis of three questionnaires 
that have been previously developed and administered by the present author. The first section of 
the paper is a brief description of each questionnaire, including tables of descriptive statistics. 

The second section is a discussion of certain aspects of questionnaire research (including the 
design of questionnaire items, collecting and analyzing the data, and writing reports). The paper 
concludes with a discussion of the validity and reliability of questionnaire research. 

Three examples of questionnaire research 
Faculty computer-assisted instruction questionnaire 

This survey was conducted to identify and prioritize the factors that influence the use of 
computer-assisted instruction (CAI) by faculty members in the course of their teaching. For this 
purpose a questionnaire was administered to the faculty of Nanyang Techuiologica! University 
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(NTU) in Singapore. The results obtained from 62 respondents indicated that the two most 
important facilitators in the use of such software were “teachers’ knowledge and skills in the CAI 
technology,” and “the availability of hardware and software”. Among inhibiting factors, “lack of 
teachers’ time,” and “lack of technical support” were the most important. Table 1 presents the 
summarized findings of this survey. 



Table 1. Facilitators and inhibitors in the faculty use of CAI (N = 62) 



Rank 


Facilitators 


M 


SD 


1 


Teacher's knowledge and skills in technology 


4.32 


0.95 


2 


Availability of hardware and software 


4.29 


1.05 


3 


Commitment by those involved in CAI 


4.26 


0.96 


4 


Availability of software information 


4.23 


0.78 


5 


Systematic planning for the use of CAI 


4.21 


0.83 


6 


Support from higher administration 


4.16 


0.96 


7 


Availability of teachers’ time 


4.13 


0.97 


8 


Availability of commercial software. 


4.11 


0.96 


9 


Universities’ formal policy for computer use 


4.03 


1.04 


10 


Availability of CAI authoring tools 


4.00 


0.94 


11 


Collaboration among developers and users 


3.98 


1.05 


12 


Support from the government. 


3.94 


0.74 


13 


Demonstration of other schools’ CAI uses. 


3.84 


0.85 


14 


Integration of CAI with schools’ goals 


3.81 


0.96 


15 


Recognition/motivation of the faculty 


3.53 


0.92 




Average 


4.06 


0.61 










Rank 


Inhibitors 


M 


SD 


1 


Lack of teachers’ time for CAI 


4.08 


0.95 


2 


Lack of technical support 


4.06 


0.94 


3 


Cost of hardware (financial resources) 


3.98 


1.11 


4 


Lack of teacher training for computer use 


3.97 


0.97 


5 


Teachers’ resistance to change 


3.92 


1.03 


6 


Lack of quality and suitable software 


3.90 


1.13 


7 


Incompatibility of teaching method with CAI 


3.89 


0.89 


8 


Lack of administrative support 


3.85 


1.02 


9 


Scheduling problems in use of computers 


3.76 


1.07 


10 


Lack of appropriate hardware 


3.74 


1.19 


11 


Lack of suitably equipped classrooms 


3.71 


1.14 


12 


Scepticism on the effectiveness of CAI 


3.56 


1.11 


13 


Lack of access to software information 


3.55 


0.99 


14 


Apprehension of teachers in teaching 


3.55 


1.04 


15 


Rapid changes in hardware and software 


3.52 


0.82 


14 


Lack of information on CAI potential 


3.45 


1.05 


17 


Lack of tests to evaluate CAI 


3.45 


0.86 


18 


Assumption that CAI needs special curricula 


3.40 


1.09 


[ Average 


3.74 


0.63 1 
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Undergraduate women’s aspirations questionnaire 

This survey was conducted to determine the status aspirations of undergraduate women at 
the University of Guam (UOG). In particular, it explored their awareness of gender equality in 
employment, and how these affect their academic and social aspirations. A random sample of 350 
women representing the culturally diverse setting of the study was selected from the degree and 
non-degree programs of UOG. The sample received the mail questionnaire, and the usable 
response rate was 32% (N = 1 1 1). The respondents indicated their strong desire for pursuing 
occupational careers and earning their own incomes. They did not perceive gender discrimination 
as being a significant impediment in employment. The results are presented in Table 2. 



Table 2. Priorities of status aspirations for undergraduate women (N = 111) 



Item 


Aspiration 


M 


SD 




How important is it for you ... 






1 


To graduate from university (your own perspective) 


4.68 


0.63* ' 


1 2 


To graduate from university (your family’s perspective) 


4.27 


0.94 


3 


To go to professional schqol_(ag^j „ 


3.23 


. u5 ; 


! 4 


To go to graduate school (master’s degree) 


3.76 


1.06 I 


5 


To go to graduate school (doctorate) 


3.12 


1.1 7_ 


6 


To become financially independent (your own perspective) 


4.47 


0.77 


7 


To become financially independent (your family’s perspective) 


4.00 


1.03 


8 


To have a prestigious occupation 


3.80 


1.14 

j 


9 


To have a profitable job (make lots of money) 


_4.01 


0.88 , 


10 


To get married in your 20s or early 30s 


3.21 


1.27 


11 


To have and raise your own child 


4.02 


1.11 




To combine home and work roles 


4.05 


0.99 


13 


To attain a position of great influence 


3.77 * 


1.09 


14 


To contribute to society through working 


4.00 


0.91 


15 


To achieve satisfaction through working 


4.43 


0.72 


16 


To marry a man with a high social standing (your own perspective) 


2.75 


1.22 


17 


To marry a man with a high social standing (your family’s perspective) 


2.84 


1.21 


18 


To have a family and have a career at the same time 


4.02 


1.04 


19 


To you that your child goes to a prestigious university 


3.57 


l.lV 


20 


To be involved in government and politics 


2.58 


1.06 


21 


To enter the business world 


2.96 


1.18 


22 


To enter the academic world 


3.62 


1.02 


23 


To work in social welfare or health care 


3.24 




24 


To engage in church affairs _ 


3.16 

1 - -• ' 


1.01 1 


25 


' To make a connection to iny^roye^qur social st^d^^ . 


3.42 

+ — — 


^ 1.08 ! 


: 26 


^ To choose friends from among those with whom you associate 


I 3.50 


[ 1.05 i 


27 


To attain status (wealth, power, prestige) 


i 325 _ . 


1 


‘ 38 


To become a mentor for the next generation^ 


’ 3.88 

' 3*16 


1.01 

+ . 


29 


To be important to society in your life 


1 1.04 ' 
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Faculty multicultural education questionnaire 

This survey was conducted to determine faculty members’ attitudes to multicultural 
education at the University of Guam, with a general focus on diversity, ethnicity, and pluralism. 
Questionnaires were sent to the entire full-time faculty through the campus mail. The usable 
return rate was 51% (N = 104). Results of the survey indicated that 71% of the respondents 
agreed with the theorized notion of pluralism — that cultural and ethnic diversity is an asset that 
enriches the learning process. The respondents, regardless of gender, age, ethnic background, or 
teaching experience, rated diversity and multiculturalism highly. The results reflected Guam’s 
ethnic and cultural diversity, and confirmed that multicultural education is an important facet of 
curriculum development in higher education. The results of this survey are presented in Table 3. 



Table 3. Rank order of five elements of teachers’ knowledge base 



Relationship skills 


M 


SD 


; N 


To be friends with someone from a different culture in Guam or anywhere 


3.76 


1.02 1 


104 


To collaborate on research and teaching with colleagues from the same cultural and 


2.81 


1.25 


103 


ethnic backgrounds as your own 








Internal consistency reliability = 0.2848 








Community knowledge 






1 


To understand or be aware of other cultures and heritage 


4.45 


0.88 


1041 


To be exposed to a culturally diversified environment 


4.19 


0.98 


r 1041 


To take the time to learn about students’ backgrounds and cultural characteristics. 


4.13 


0.88 


i 104 


To respect and accommodate students’ individual and culture-based learning styles 


4.04 


0.90 


1 104 


Internal consistency reliability = 0.7724. 








Empathy 




1 

1 . 1 


1 To support the academic success of students from different cultural and ethnic 


4.57 


0.80' 104 1 


backgrounds than your own 








To challenge and avoid using stereotypes in your teaching 


4~45 


0.94 1 


104 1 


To become a culturally sensitive and responsive teacher 


4.52 


0.78 


103 


To become informed about cultural and ethnic differences 


4.29 


0.81 


104 


(Internal consistency reliability = 0.7935) 








Cultural conflicts 








To eradicate prejudice in your professional life 


4.71 


_o,7r 


I 104 


f — ' • — ^ 

1 To eradicate preiudice in your personal life 


4.64 


0.79' 


104' 


1 To employ Eurocentric pedagogy in your teaching. 


3.30 


1.00 


lOl" 


1 To associated with people from the same cultural and ethnic backgrounds as your own 


3.21 


1.15 


r 104 


1 Internal consistency reliability = 0.5078 






1 


! Relevant curriculum 








1 To provide an environment for the free and open expression of ideas and beliefs 


4.62 


0.72 1 104 


To use culturally relevant examples and materials in teaching 


j_ 4.33 


0.85 


104 ■ 


To integrate multicultural perspectives in your teaching 


4.22 


1.01 


104 


To be exposed to a culturally diversified environment 


i 4.19 


0.98 


102 



: (Internal consistency reliability = 0.7640) 
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Designing, analyzing, and reporting questionnaire research 

Fink (1995) has identified the following six requirements for a useful questionnaire survey: 

• Specific, measurable objectives; 

• Sound research design; 

• Sound choice of population or sample; 

• Reliable and valid instruments; 

• Appropriate analysis; and 

• Accurate reporting of the results. 

Each of these is considered below in relation to the surveys under consideration in the present 
paper. 

Specific, measurable objectives 

With respect to the first criterion noted above, each of the three questionnaires presented here 
had specific, measurable objectives. Preparing such objectives is the first step in good 
questionnaire research, and Patten (2001) has made the following pertinent observations in this 
regard: (i) if the objective is too broad, it is difficult to provide questionnaire items that are 
effective and relevant; (ii) the objectives, as developed, should be reviewed by external experts 
before proceeding; and (iii) the literature relevant to the objective should be thoroughly reviewed. 

Sound research design 

With respect to the second criterion noted above (sound research design), Fink (1995, p. 3) 
has observed that a survey design is “a way of arranging the environment in which a survey takes 
place.” The three questionnaires discussed in the present paper all used a cross-sectional type of 
design, and were relatively simple (ranging from three to six pages). In designing the 
questionnaire, account was taken of the basic question of the feasibility of administering the 
questionnaire to the particular population of interest, and the questionnaire design was modified 
appropriately. 

A Likert-type was used in all three questionnaires under discussion in the present paper. 
Likert scales are the most frequently used tools in questionnaires of this type. They allow 
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participants to respond to each question in terms of five degrees of agreement (usually ranging 
from 1 = “strongly disagree” to 5 = “strongly agree”). In these questionnaires, each response was 
given a numerical score to reflect weighting, and the scores were totalled to measure the 
respondent’s attitude or perception. In passing, it should be noted that in designing Likert-type 
scales, Patten (2001) has emphasized that it is better to use terms such as “almost never” and 
“almost always” for item choices, because few things in life are definitively “never” or “always.” 

Sound choice of population or sample 

With respect to the third criterion noted above (sampling), for the faculty multicultural 
education questionnaire, the population and sample of the study was the entire UOG faculty. In 
the case of the faculty computer-assisted instruction questionnaire, the division of Education had 
59 faculty members (40% male). Of 208 faculty members in Business, 75% were male. All 
members of the faculty of Education received the questionnaire. To match this sample size, 59 
faculty members of Business were randomly selected. A random sampling procedure was used 
for the undergraduate women aspirations questionnaire; and the sample was fairly representative 
of the population (see Table 4). 



Table 4.1 Undergraduate enrolmen t (1998 Spring Semester) 

Sex Number Percentage 

Male 1,374 (39%) 

Female 2,146 61% 

Total 3,520 100% 



Table 4.2 Population and sample comparison 





Population 


Sample 


Ethnic Background 


Chamorro 


918 (42.8%) 


49(44.1%) 


Asian Philippines 


663 (30.9%) 


42 (37.8%) 


Asian Others 


192 (8.9%) 


10(9.0%) 


Micronesian 


111 (5.2%) 


3 (2.7%) 


“Stateside” American 


190 (8.9%) 


3 (2.7%) 


Other 


72 (3.4%) 


4 (3.6%) 


Total 


2,146 (100%) 


1 1 1 (100.0%) 


Age 


21 or less 


753 (25.1%) 


45 (40.5%) 


22-24 


499 (23.3%) 


29 (26.1%) 


25-27 


269(12.5%) 


12(10.8%) 


28-30 


154 (7.2%) 


4 (3.6%) 


3 1 or over 


470 (21.9%) 


21 (18.9%) 


Total 


2,146(100%) 


111 (100%) 
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Reliable and valid instruments 

With respect to the fourth criterion noted above (reliable and valid instruments), a 
questionnaire should contain items that are pertinent to the objectives. “Valid data come from 
surveys that measure what they purport to measure” (Fink, 1995, p. 5). The three questionnaires 
under consideration here were all reviewed by colleagues and then pilot-tested to ascertain that 
the questionnaire items were clearly understood by the participants. 

In this respect, the general guidelines for developing survey instruments as advanced by 
Wiersma (2000, p. 169) are useful: 

(i) except for possibly a few items that request background or demographic information, the 
item should relate directly to the research problem, questions, or hypotheses; 

(ii) items should be clear and unambiguous; 

(iii) only one concept should be included in a single item; 

(iv) the use of leading questions should be avoided; 

(v) only information that the respondent is able to provide should be requested; and 

(vi) shorter items are to be preferred to longer items, and simpler items are to be preferred to 
complex items. 

Appropriate analysis 

With respect to the fifth criterion noted above (appropriate analysis), although surveys can 
use a variety of conventional statistical procedures to analyze the data, the appropriate analysis 
depends on the survey aims (such as description, comparison, and correlation) and the size of the 
sample. 

In comparing education and business faculty members in the first questionnaire, t-test (at an 
alpha level of .05) was used to determine the differences for all factors between the two groups in 
assessing facilitators and inhibitors in the use of CAI. Similarly, in the second questionnaire, t- 
test was used (again with an alpha level of .05) in making the comparison between Chamorro and 
non-Chamorro women. The reliability coefficient alpha across all the 29 aspiration items was 
.8677. In the third survey, the 18 item scores were combined into five cluster scores. Internal 
consistency reliability was estimated by computing alpha coefficients for the clusters. These 
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ranged from .28 (relationship skills) to .79 (empathy). It should be noted that a coefficient alpha 
of .28 is unreliable, indicating in this case that items in relationship skills were not measuring the 
same thing. ANOVA was used to examine the significant differences on the variables of gender, 
age, ethnicity, and teaching experience in each of the elements (at an alpha level of .01). 

Accurate reporting of the results 

With respect to the sixth criterion noted above (accurate reporting of results), Fink (1995) has 
noted that accurate survey reports require knowledge of how to use tables and figures to present 
information. 

In all the three questionnaires under consideration here, both ordinal data and nominal data 
(that is, categorical data such as age and gender) were summarized in tables. Tables are useful and 
effective in most cases, although it should be noted in passing that Patten (1995, p. 90) is of the 
opinion : “Generally, figures are more eye-catching, and many people find them easier to 
interpret than tables.” 

Overall assessment of the surveys 



Table 5 is a summary of the three questionnaires from objectives to results. 



Table 5. Summary of three questionnaire studies 





Faculty computer-assisted 
instruction questionnaire 
(1995) 


Undergraduate women’s 
aspirations questionnaire 
(1998) 


Faculty multicultural 
education questionnaire 
(1999) 


Objective 


To identify and prioritise the 
factors influencing the 
university faculty members' 
use of computer-assisted 
instruction (CAI) in 
Singapore. 


To determine UOG 
undergraduate women’s 
views on quality of life, 
identifying their educational 
and social status aspirations 


To examine UOG faculty 
members’ attitudes to 
diversity and multicultural 
education, with a focus on 
diversity, ethnicity, and 
pluralism. 


Instrument 

! 

H 

it 

i — .......j 


A 3-page questionnaire 
(incorporating demographic 
information) with faculty 
members being asked to 
indicate on a 5-point scale 
the Importance of 33 factors 
(15 facilitators; 18 
inhibitors) in the use of 
CAI. 

1 


A 6-page questionnaire 
(incorporating demographic 
information) with women 
being asked to indicate on a 
5-point scale the importance 
of 29 aspiration items, 15 
gender equality items, and 8 
self-evaluation items. 

1 


A 3-page questionnaire 
(incorporating demographic 
information) with faculty 
members being asked to 
indicate on a 5-point scale 
the importance of 18 
diversity and | 

j multiculturalism questions, i 

L i 



O 
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Sample 


Participants were drawn from 
two groups (business and 
education). A survey was 
sent by campus mail to 118 
faculty members and the 
usable response rate was 
53% (N = 63). 


The study population was the 
entire undergraduate women 
at UOG. A survey was 
mailed to randomly selected 
350 women: the usable 
response rate was 32% (N = 
111). 


The study population was the 
entire fiill-time faculty at 
UOG. A survey was sent to 
all faculty members by 
campus mail: the usable 
response rate was 51% (N 
= 104). 


Data analysis 


In prioritizing items, overall 
mean scores and standard 
deviations for all 
respondents by all items 
were calculated, and Mest 
was used to determine the 
differences between these 
two groups. 


In prioritizing items, overall 
mean scores and standard 
deviations for all 
respondents by all items 
were calculated, and r-test 
was used to determine the 
differences between 
Chamorro and non- 
Chamorro women. 


ANOVA was used to 
examine if the obtained 
sample and potential 
sample differed 
significantly on the 
variables of gender, age, 
ethnicity, and teaching 
experience in each of the 
elements. 


Results 


Major facilitators were 
“teachers’ knowledge and 
skills in technology” and 
“availability of 
hardware/software”, whereas 
“lack of teachers’ time” and 
“lack of technical support” 
were the most important 
inhibitors for the use of 
CAI. 


Respondents indicated their 
strong desires to achieve 
self-satisfaction through 
pursuing occupational 
careers and earning their 
own incomes. There were 
no significant differences 
between the two groups 
(Chamorro and non- 
Chamorro) in aspirations. 


71% of respondents agreed 
with pluralism applied to 
education. Regardless of 
gender, age, ethnic 
background, or teaching 
experience, the respondents 
rated diversity and 
multiculturalism highly. 



Improving reliability and validity in questionnaire research 

Why do people use questionnaires? Questionnaires are a practical method for obtaining many 
types of information from people and, in many circumstances, the most economical method. A 
common misconception is that it is easy to design and conduct a survey questionnaire. However, 
as Litwin (1995, p. 1) has noted: “ ... there are good surveys and bad ones ... good surveys yield 
critical information and provide important windows into the heart of the topic of interest.” To 
improve future studies, it is necessary to discuss the major limitations shown by the above three 
examples of questionnaire research. 

The first limitation to be considered in assessing the three survey instruments discussed in 
the present paper is the question of their reliability, validity, and practicality. Although the 
instruments were revised as a result of expert opinion, and although they were pilot tested, they 
might still not have been as valid (accurate) and as reliable (consistent) as standardized or 
published instruments. To achieve reliability, test-retest reliability should be applied (that is. 
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having the same respondents complete a survey at two different points in time). Or alternate- 
form reliability should be performed (that is, using differently worded items or changing the order 
of items to measure the same attribute). These tests are useful, even though, in practice, their 
usefulness can be limited by the fact that people might “become familiar with the items and 
simply answer based on their memory of what they answered the last time” (Litwin, 1995, p. 

13). Expert opinion can be used to ensure that questionnaires adequately cover the relevant 
topics, and to judge how well the instrument meets the standard. 

The second limitation to be considered is the exploratory nature of these studies. In the first 
two questionnaires, single-item comparisons were made, rather than multiple item measures. 
Questionnaires containing single items are easier to develop and quicker to administer. However, 
as Litwin, 1995, p. 21) has observed: “ . . . the data set is richer and more reliable if several 
different items are used to gain information about a particular behavior or topic” (Litwin, 1995, p. 
21). The third questionnaire used multiple items, which was more reliable when analyzing data in 
relation to the specific objectives of the research. 

The third limitation to be considered is the Likert-scale method. A 7-point or 9-point scale 
might have yielded more accurate results than the 5-point scale used in Tables 1 to 3, because the 
mean scores suggested there was no clear distinction between the importance of each item. 

Finally, Wiersma (2000) has noted that a covering letter is an essential part of any 
questionnaire. A covering letter introduces individuals to the purpose and nature of the 
questionnaire and motivates a response. Figure 1 depicts the covering letter used with the faculty 
multicultural education questionnaire. In regard to recipients completing the questionnaire in an 
accurate and timely manner, Wiersma's (2000) following observations are useful: the purpose 

of the questionnaire is clearly stated, confidentiality is assured, a deadline is given for the return 
of the questionnaire, and appreciation for completing the questionnaire is expressed” (p. 173). 
Indeed, without respondents, questionnaire analysis is impossible and the cover letter is the 
pS.SS3.^C tC the successful questionnaire research. 
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Figure 1. Cover letter used with the faculty multicultural education questionnaire 



Dear UOG Faculty: 

The University of Guam and the region of Micronesia lie between East and West, and one of the 
paramount goals of this institution is to provide undergraduate and graduate programs that are not 
only international in context but also build upon this region’s unique and varied cultural experiences. 

We respectfully ask you to take a few minutes out of your busy schedules to complete the attached 
survey concerned with diversity and multiculturalism. By answering this survey questionnaire, you 
are supporting faculty research and contributing to the pursuit of knowledge. Our research findings 
will be summarized and communicated back to you. All efforts have been taken to maintain 
confidentiality and we assure all participants that the data will be presented only in summary form 
and that the individual surveys themselves will be kept secure by us. 

Please complete and return the survey by November 12, 1999, to ensure that your comments are 
included in our survey. Please drop it in the interoffice mail. Again, thank you for your 
participation. 

Sincerely, 



Yukiko Inoue 
College of Education 

Kirk Johnson 

College of Arts and Sciences 
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